Search for: All records

Transformer models have revolutionized machine learning, yet the underpinnings behind their success are only beginning to be understood. In this work, we analyze transformers through the geometry of attention maps, treating them as weighted graphs and focusing on Ricci curvature, a metric linked to spectral properties and system robustness. We prove that lower Ricci curvature, indicating lower system robustness, leads to faster convergence of gradient descent during training. We also show that a higher frequency of positive curvature values enhances robustness, revealing a trade-off between performance and robustness. Building on this, we propose a regularization method to adjust the curvature distribution and provide experimental results supporting our theoretical predictions while offering insights into ways to improve transformer training and robustness. The geometric perspective provided in our paper offers a versatile framework for both understanding and improving the behavior of transformers. 

Maternal trauma influences infant and adult health outcomes and may impact future generations through epigenetic modifications such as DNA methylation (DNAm). Research in humans on the intergenerational epigenetic transmission of trauma effects is limited. In this study, we assessed DNAm signatures of war-related violence by comparing germline, prenatal, and direct exposures to violence across three generations of Syrian refugees. We compared families in which a pregnant grandmother versus a pregnant mother was exposed to violence and included a control group with no exposure to war. We collected buccal swab samples and survey data from mothers and 1-2 children in each of 48 families (n = 131 participants). Based on an epigenome-wide association study (EWAS), we identified differentially methylated regions (DMPs): 14 were associated with germline and 21 with direct exposure to violence. Most DMPs showed the same directionality in DNAm change across germline, prenatal, and direct exposures, suggesting a common epigenetic response to violence. Additionally, we identified epigenetic age acceleration in association with prenatal exposure to violence in children, highlighting the critical period of in utero development. This is the first report of an intergenerational epigenetic signature of violence, which has important implications for understanding the inheritance of trauma.